Information and Entropy

The general problem of inductive inference is to update from a prior probability distribution to a posterior distribution when new information becomes available. Bayes' rule is the natural way to update when the new information is in the form of data while Jaynes’ method of maximum entropy, MaxEnt, is designed to handle information in the form of constraints. However, the range of applicability of either method is limited: Bayes' rule can handle arbitrary priors and data, but not arbitrary constraints, and MaxEnt can handle arbitrary constraints (including data) but not arbitrary priors. We show that Skilling's method of induction leads to a unique general theory of inductive inference, the method of Maximum relative Entropy (M.E.). The M.E. method is designed for updating from arbitrary priors given information in the form of arbitrary constraints (including data). Four axioms (locality, coordinate invariance, consistency for independent subsystems, and consistency for large numbers) suffice to single out the logarithmic relative entropy as the unique tool for updating; other entropy functionals are ruled out although they might still be useful for other purposes. The M.E. method includes both MaxEnt and Bayes' rule as special cases and therefore it unifies the two themes of these workshops – the Maximum Entropy and the Bayesian methods – into a single general inference scheme that allows us to handle problems that lie beyond the reach of either of the two methods separately. I conclude with a couple of simple illustrative examples.